The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks

Brains process information in spiking neural networks. Their intricate connections shape the diverse functions these networks perform. In comparison, the functional capabilities of models of spiking networks are still rudimentary. This shortcoming is mainly due to the lack of insight and practical algorithms to construct the necessary connectivity. Any such algorithm typically attempts to build networks by iteratively reducing the error compared to a desired output. But assigning credit to hidden units in multi-layered spiking networks has remained challenging due to the non-differentiable nonlinearity of spikes. To avoid this issue, one can employ surrogate gradients to discover the required connectivity in spiking network models. However, the choice of a surrogate is not unique, raising the question of how its implementation influences the effectiveness of the method. Here, we use numerical simulations to systematically study how essential design parameters of surrogate gradients impact learning performance on a range of classification problems. We show that surrogate gradient learning is robust to different shapes of underlying surrogate derivatives, but the choice of the derivative’s scale can substantially affect learning performance. When we combine surrogate gradients with a suitable activity regularization technique, robust information processing can be achieved in spiking networks even at the sparse activity limit. Our study provides a systematic account of the remarkable robustness of surrogate gradient learning and serves as a practical guide to model functional spiking neural networks

Talking science, online

Traditional scientific conferences and seminar events have been hugely disrupted by the COVID-19 pandemic, paving the way for virtual forms of scientific communication to take hold and be put to the test

Nonnormal amplification in random balanced neuronal networks

In dynamical models of cortical networks, the recurrent connectivity can amplify the input given to the network in two distinct ways. One is induced by the presence of near-critical eigenvalues in the connectivity matrix W, producing large but slow activity fluctuations along the corresponding eigenvectors (dynamical slowing). The other relies on W being nonnormal, which allows the network activity to make large but fast excursions along specific directions. Here we investigate the tradeoff between nonnormal amplification and dynamical slowing in the spontaneous activity of large random neuronal networks composed of excitatory and inhibitory neurons. We use a Schur decomposition of W to separate the two amplification mechanisms. Assuming linear stochastic dynamics, we derive an exact expression for the expected amount of purely nonnormal amplification. We find that amplification is very limited if dynamical slowing must be kept weak. We conclude that, to achieve strong transient amplification with little slowing, the connectivity must be structured. We show that unidirectional connections between neurons of the same type together with reciprocal connections between neurons of different types, allow for amplification already in the fast dynamical regime. Finally, our results also shed light on the differences between balanced networks in which inhibition exactly cancels excitation, and those where inhibition dominates.Comment: 13 pages, 7 figure

Developmental depression-to-facilitation shift controls excitation-inhibition balance

Changes in the short-term dynamics of excitatory synapses over development have been observed throughout cortex, but their purpose and consequences remain unclear. Here, we propose that developmental changes in synaptic dynamics buffer the effect of slow inhibitory long-term plasticity, allowing for continuously stable neural activity. Using computational modeling we demonstrate that early in development excitatory short-term depression quickly stabilises neural activity, even in the face of strong, unbalanced excitation. We introduce a model of the commonly observed developmental shift from depression to facilitation and show that neural activity remains stable throughout development, while inhibitory synaptic plasticity slowly balances excitation, consistent with experimental observations. Our model predicts changes in the input responses from phasic to phasic-and-tonic and more precise spike timings. We also observe a gradual emergence of short-lasting memory traces governed by short-term plasticity development. We conclude that the developmental depression-to-facilitation shift may control excitation-inhibition balance throughout development with important functional consequences

Percutaneous reduction and fixation of intraarticular calcaneal fractures

Objective: Percutaneous reduction by distraction and subsequent percutaneous screw fixation to restore calcaneal and posterior talocalcaneal facet anatomy. The aim of this technique is to improve functional outcome and to diminish the rate of secondary posttraumatic arthrosis compared to conservative treatment and, secondly, to reduce infectious complications compared to open reduction and internal fixation (ORIF). Indications: Sanders type II-IV displaced intraarticular calcaneal fractures. Contraindications: Isolated centrally depressed fragment. Contraindications: Patients who are expected to be noncompliant. Surgical Technique: Four distractors (Synthes™) are positioned, two on each side of the foot, between the tuberosity of the calcaneus and talus and between the tuberosity and cuboid. A distracting force is given over all four distractors. A blunt drifter is then introduced from the plantar side to unlock and push up any remaining depressed parts of the subtalar joint surface of the calcaneus. The reduction is fixated with two or three screws inserted percutaneously. Postoperative Management: Directly postoperatively, full active range of motion exercises of the ankle joint can start, with the foot elevated in the 1st postoperative week. Stitches are removed after 14 days. Implant removal is necessary in 50-60% of patients. Results: Between 1999 and 2004, 59 patients with 71 fractures were treated by percutaneous skeletal triangular distraction and percutaneous fixation. A total of 50 patients with 61 fractures and a minimum follow-up of 1 year were available for follow-up. According to the American Orthopaedic Foot and Ankle Society Hindfoot Score, 72% had a good to excellent result. A secondary subtalar arthrodesis was performed in five patients and planned in four (total 15%). Böhler's angle increased by about 20° postoperatively. Sagittal motion was 90% andsubtalar motion 70% compared to the healthy foot

Connection-type-specific biases make uniform random network models consistent with cortical recordings

Uniform random sparse network architectures are ubiquitous in computational neuroscience, but the implicit hypothesis that they are a good representation of real neuronal networks has been met with skepticism. Here we used two experimental data sets, a study of triplet connectivity statistics and a data set measuring neuronal responses to channelrhodopsin stimuli, to evaluate the fidelity of thousands of model networks. Network architectures comprised three neuron types (excitatory, fast spiking, and nonfast spiking inhibitory) and were created from a set of rules that govern the statistics of the resulting connection types. In a high-dimensional parameter scan, we varied the degree distributions (i.e., how many cells each neuron connects with) and the synaptic weight correlations of synapses from or onto the same neuron. These variations converted initially uniform random and homogeneously connected networks, in which every neuron sent and received equal numbers of synapses with equal synaptic strength distributions, to highly heterogeneous networks in which the number of synapses per neuron, as well as average synaptic strength of synapses from or to a neuron were variable. By evaluating the impact of each variable on the network structure and dynamics, and their similarity to the experimental data, we could falsify the uniform random sparse connectivity hypothesis for 7 of 36 connectivity parameters, but we also confirmed the hypothesis in 8 cases. Twenty-one parameters had no substantial impact on the results of the test protocols we used

Neural mechanisms of attending to items in working memory

Working memory, the ability to keep recently accessed information available for immediate manipulation, has been proposed to rely on two mechanisms that appear difficult to reconcile: self-sustained neural firing, or the opposite-activity-silent synaptic traces. Here we review and contrast models of these two mechanisms, and then show that both phenomena can co-exist within a unified system in which neurons hold information in both activity and synapses. Rapid plasticity in flexibly-coding neurons allows features to be bound together into objects, with an important emergent property being the focus of attention. One memory item is held by persistent activity in an attended or "focused" state, and is thus remembered better than other items. Other, previously attended items can remain in memory but in the background, encoded in activity-silent synaptic traces. This dual functional architecture provides a unified common mechanism accounting for a diversity of perplexing attention and memory effects that have been hitherto difficult to explain in a single theoretical framework

Training deep neural density estimators to identify mechanistic models of neural dynamics

Mechanistic modeling in neuroscience aims to explain observed phenomena in terms of underlying causes. However, determining which model parameters agree with complex and stochastic neural data presents a significant challenge. We address this challenge with a machine learning tool which uses deep neural density estimators—trained using model simulations—to carry out Bayesian inference and retrieve the full space of parameters compatible with raw data or selected data features. Our method is scalable in parameters and data features and can rapidly analyze new data after initial training. We demonstrate the power and flexibility of our approach on receptive fields, ion channels, and Hodgkin–Huxley models. We also characterize the space of circuit configurations giving rise to rhythmic activity in the crustacean stomatogastric ganglion, and use these results to derive hypotheses for underlying compensation mechanisms. Our approach will help close the gap between data-driven and theory-driven models of neural dynamics

Impact of Gut Bacteria on the Infection and Transmission of Pathogenic Arboviruses by Biting Midges and Mosquitoes

Tripartite interactions among insect vectors, midgut bacteria, and viruses may determine the ability of insects to transmit pathogenic arboviruses. Here, we investigated the impact of gut bacteria on the susceptibility of Culicoides nubeculosus and Culicoides sonorensis biting midges for Schmallenberg virus, and of Aedes aegypti mosquitoes for Zika and chikungunya viruses. Gut bacteria were manipulated by treating the adult insects with antibiotics. The gut bacterial communities were investigated using Illumina MiSeq sequencing of 16S rRNA, and susceptibility to arbovirus infection was tested by feeding insects with an infectious blood meal. Antibiotic treatment led to changes in gut bacteria for all insects. Interestingly, the gut bacterial composition of untreated Ae. aegypti and C. nubeculosus showed Asaia as the dominant genus, which was drastically reduced after antibiotic treatment. Furthermore, antibiotic treatment resulted in relatively more Delftia bacteria in both biting midge species, but not in mosquitoes. Antibiotic treatment and subsequent changes in gut bacterial communities were associated with a significant, 1.8-fold increased infection rate of C. nubeculosus with Schmallenberg virus, but not for C. sonorensis. We did not find any changes in infection rates for Ae. aegypti mosquitoes with Zika or chikungunya virus. We conclude that resident gut bacteria may dampen arbovirus transmission in biting midges, but not so in mosquitoes. Use of antimicrobial compounds at livestock farms might therefore have an unexpected contradictory effect on the health of animals, by increasing the transmission of viral pathogens by biting midges.</p